In enterprise class Java applications, such as WebSphere Commerce, the execution profile typically includes a large number of methods; each of the methods takes a very small proportion of overall execution time. However, over time, these methods get compiled simply because the initial compilation is driven by method invocation counts. Once a method is executed certain times, the method gets compiled regardless of how long it has taken to execute the method for the certain times. This means that the Just-In-Time (JIT) compiler's code cache can contain many methods that are executed only infrequently. Such compilations can increase (code memory) footprints and startup/ramp-up time with little beneficial effect on throughput. However, the real problem stems from the fact that there is only a fixed amount of a code cache available for each instance of the JIT compiler. If the code cache gets filled with methods less important for throughput performance, more important subsequent compilations may not be successful since there is no space to write the generated code. Interpreting the more frequently executed methods which can not be compiled has a noticeable negative impact on throughput, because interpreting is an order of magnitude (e.g., 10×) slower than compiled code.
The simplest approach to a bounded code cache is to flush its entire content when the bounded code cache becomes full. This solution, referred to as code pitching, is employed by Microsoft's .NET Compact Framework as described by Anthony, Leung, and Srisa-an (“To JIT or not to JIT: The Effect of Code Pitching on the Performance of .NET Framework”, 2005). As shown by the authors, while code pitching can degrade runtime performance, if it is done rarely enough, the penalty is barely visible. Thus, picking an appropriate maximum size for the code cache is important. One drawback of the study, as admitted by the authors, is that they considered only tiny applications which are not representative of a real-life workload. For big applications, where compilation overhead is significant, simply throwing away all the compiled code is going to put unnecessary strains on the computational resources.
Zhang and Krintz (“Profile-Driven Code Unloading for Resource-Constrained JVMs”, 2004) propose to periodically (e.g., every 10 ms) discard, from the code cache, compiled methods which are unlikely to be used in the future. The candidates for eviction are chosen with the help of a lightweight profiling mechanism based on sampling. In a follow-up work (“Adaptive Code Unloading for Resource-Constrained JVMs”, 2004), the authors develop an adaptive mechanism which determines when method eviction should take place. The idea is that frequency of the eviction should be dictated by memory pressure: the more time the JVM spends in the garbage collector, the more frequent the eviction process should be. However, this conclusion is a direct consequence of their particular implementation: code fragments reside in the Java heap and therefore compete for space with Java objects. Freeing more code makes room for Java objects and improves the time spent in the garbage collector. An important disadvantage of the selective method eviction process is that it creates a lot of fragmentation.
Hazelwood and Smith (“Code Cache Management Schemes for Dynamic Optimizers”, 2002) avoid fragmentation by evicting code fragments in a FIFO (First In First Out) manner. The code cache is considered a big circular buffer and eviction takes place on demand, when a new code fragment needs to be stored. While this approach simplifies code cache management, it brings in the previous problem: because it is oblivious to the frequency of execution of code fragments, important compiled pieces of code can be evicted from the cache. However, the negative effects are likely milder than in the code pitching case where the entire cache is cleared.
Hazelwood and Smith (“Generational Cache Management of Code Traces in Dynamic Optimization Systems”, 2003) propose the use of generational code caches. The code cache space is split into three regions: nursery, probation cache, and persistent cache. New code fragments are stored in the nursery cache, possibly evicting other code fragments in case there is not enough room. The victim of an eviction is moved into the probation cache whose main role is to determine how important the evicted code fragments are. When the room needs to be made into the probation cache, either the code fragment to be eliminated is moved into the persistent code cache if it has been used since moved to the probation cache, or the code fragment to be eliminated is simply discarded. This technique works relatively well, but complicates code cache management a great deal because of the following factors. (1) Moving code fragments around from one code cache to another requires the code to be relocatable. (2) Direct transition from one code fragment to another is not possible, unless metadata is kept around to fix all “callers” when the “callee” is moved or discarded (it is called “link repair” by the authors of the paper). (3) A mechanism that monitors code usage is required to determine whether or not a piece of code should be moved to the persistent code cache or discarded.
Throughput and garbage collection (GC) pause times can also be indirectly affected by the caching of a large number of infrequently used compiled methods. A GC cycle is often the perfect time to reclaim the code cache space for unloaded or recompiled methods; the reclamation process includes locating and freeing up the run time metadata (e.g., class hierarchy assumptions, GC relevant stack/registers information, exception tables, etc.). Different kinds of JIT metadata are stored in different hashtables (for example, in IBM J9 VM or IBM Testarossa JIT compiler) and hence locating all the information can be a very time consuming process that scales linearly with the amount of generated code. Even if the compilations are only infrequently used, the amount of metadata generated for these compilations can still affect the overall time for locating JIT metadata.